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(57) Abstract: The idea of the invention is to produce an optical flow uniquely determining points of correspondence for points 
between original images on the surface of the original images. The flow is determined in such a way that all of the new images 
to be formed between the original images can be formed by means of the same optical flow. Each motion vector of the flow has a 
permitted area of movement defined by the morion vectors surrounding said motion vector, within the bounds of which its direction 
can move when the flow is formed. The area of movement is used as an additional constraint the satisfying of which will ensure 
the uniqueness of the flow. A unique flow is realized by adjusting the proportion of cross-correlation taking account of motion and 
of the smoothness criterion when the flow is formed, in such a way that in the vicinity of violation of the additional constraints, 
the proportion of cross-correlation taking account of motion is suppressed. This should preferably be done in such a way that the 
proportion of the cross-correlation is zeroed when the additional constraints are violated. 
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Optical flow and image forming 



Field of the Invention 

The present invention relates generally to the field of digital image 
5 processing and, more particularly, to recognizing and analyzing movement in 
images. The invention also relates to measuring of the optical flow between 
two images and to forming of new pictures that are intermediate forms of two 
originals in their features. Optical flow denotes a field formed from motion 
vectors, connecting the corresponding image points of two pictures. The 
10 present invention can also be utilized to enhance image resolution, i.e. the 
pixel number. 



Background of the Invention 

Digital visual imaging provides versatile possibilities for processing 
15 and compressing pictures and forming new visual images. Owing to the 
movements of the object, there usually are differences between two 
successive images. Digital visual imaging can also be used for motion 
recognition. The application area of visual imaging is extensive: a wide 
variety of computer vision applications, compression of video images, clinical 

20 section images, enhancement of resolution, etc. 

One of the most usual functions of visual imaging is the 
representation of motion information between two pictures (usually taken in 
successive moments). For example, if the first picture depicts a dog wagging 
its tail, in the second picture the position of the dog's tail is different. The aim 

25 is to determine the motion information included in the movement of the tail. 
This is done by defining the optical flow between the pictures, that is, the 
motion vector field that connects the corresponding points in the two images 
(a and b, Figure 1). When the optical *!ow has been defined, it can be used 
to form a new picture - a so-called intermediate picture - between the two 

30 originals. 

There are many prior art methods for defining optical flow, for 
example gradient-based methods and motion estimation over a hierarchy of 
resolutions. Other known methods include block-based methods and 
methods matching objects in images (U.S. Patent 5,726,713 and U.S. Patent 
35 4,796,087). These methods are not, however, capable of yielding a similar 
high-resolution flow as the gradient-based methods. The so-called feature- 
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based methods, founded on recognizing and connecting the corresponding 
features between two images, are in a category by itself. Yet it is difficult to 
develop any general feature classification, wherefore the feasibility of 
feature-based methods is largely dependent on the picture material to be 
5 processed. 

The gradient-based methods (U.S. Patent 5,241,68) aim at forming 
a motion vector field between two images, connecting the images in such a 
way that the square sum of the differences in value (e.g. differences in 
luminosity) of the combined image points is minimized (minimising of cross- 
10 correlation taking account of motion). However, this condition is not sufficient 
to uniquely determine the flow generated: the smoothness of the flow, i.e. the 
requirement that the distance between the points in the first picture must be 
approximately the same in the second picture as well, is used as an 
additional condition. 

15 The smoothness criterion for the flow will present problems at so- 

called occlusion points, where the object present in a picture moves to be 
located on top of another object in the picture (Figure 2, reference o). At 
these points, the optical flow is not smooth. It has been attempted to avoid 
the problems caused thereby by relinquishing the smoothness criterion at 

20 points that can be presumed to be the edge points of an object represented 
in an image. For example rapid changes in the gray scale of the images and 
the properties of the flow defined can be used in classifying the points as 
likely edge points. 

Owing to the non-smoothness of the optical flow, the prior art 

25 methods must calculate all the intermediate pictures separately, which will 
increase the requisite computation and the time required for the process, 
depending on the number of intermediate images required. The aim of the 
present invention is to diminish these disadvantages of the prior art. 

30 Summary of the Invention 

The object of the invention is achieved in the way set forth in the 
claims. The idea of the invention is to produce a flow that uniquely 
determines the points in the intermediate images corresponding to the points 
on the surfaces of the original images, in other words, each point in an inter- 
35 mediate image has one corresponding point in both of the original pictures. 
The flow is determined in such a way that all of the new pictures to be 
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formed in between the original pictures can be formed by means of the same 
optical flow. Each motion vector of the flow has a permitted area of 
movement defined by the motion vectors surrounding said motion vector, 
within the bounds of which its direction can change when the flow is formed. 
5 Hence, the area of movement is applied as an additional constraint in the 
present invention, adherence to which ensures the uniqueness of the flow. 
Since the image information often includes areas (such as occlusion points) 
infringing the additional constraint, ensuring the uniqueness of these areas 
will warrant the uniqueness of the entire flow. An unique flow can be realized 

10 by adjusting the proportion of the cross-correlation taking account of 
movement and of the smoothness criterion when the flow is formed, in such 
a way that in the vicinity of violation of the additional constraints, the 
proportion of cross-correlation taking account of movement is suppressed. 
This should preferably be done in such a way that the proportion of cross- 

15 correlation is zeroed when the additional constraints are violated. 

Since in the present method an unlimited number of new images 
can be formed by means of one flow, the method enables, among other 
things, a more efficient way than heretofore of compressing an image set. 
The invention enables the use of a rapidly convergent and computationally 

20 light algorithm for determining the flow, which makes it advantageous to 
utilize the invention also in applications to which the production of new 
images does not directly relate. Furthermore, the invention enables, for 
instance, enlargement of an original image, so that the enlarged image is 
more agreeable to the human eye than an image enlarged by the prior art 

25 methods. 

List of Drawings 

In the following, the invention will be described in detail with 
reference to the examples of Figures 1-15, set forth in the accompanying 
30 drawings in which 

Fig. 1 shows an example of an optical flow between two images, 
Fig. 2 shows an example of occlusion points of an optical flow, 
Fig. 3 illustrates the determining of an individual motion vector of 
the flow in accordance with the invention and the shift of the motion vector, 
35 Fig. 4 depicts an example of the search of a motion vector passing 

through an image point in a new image in accordance with the invention, 
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Fig. 5 illustrates in flow chart form the search of the value of an 
image point in a new image, 

Fig. 6 illustrates the change in the weighting factor of the cross- 
correlation portion in the permitted area of movement of the motion vector, 
5 Fig. 7 illustrates the permitted area of an individual motion vector 

on the image plane when the constraining factor is a single neighbouring 
motion vector, 

Fig. 8 illustrates the permitted area of an individual motion vector 
on the image plane when the constraining factor is constituted by four 
10 neighbouring motion vectors, 

Fig. 9 illustrates the permitted areas of an individual motion vector 
on both image planes, 

Fig. 10 depicts in flow chart form an example of forming an optical 
flow in accordance with the invention, 
15 Fig. 11 depicts in flow chart form an example of forming a new 

image on a new image plane, 

Fig. 12 shows by way of example enlarging of an aliased image in 
by prior art methods and by the method of the invention, 

Fig. 13 shows an example of motion vectors to be formed between 
20 two pixel lines/columns, 

Fig. 14a depicts in flow chart form an example of forming a new 
enlarged image in accordance with the invention, 

Fig. 14b represents a continuation of Fig. 14a, 

Fig. 15 illustrates image enlargement in accordance with the 
25 invention when the pixel lines and columns are processed simultaneously. 

Detailed Description of the Invention 

In the following, the mathematical background of the invention will 
be described. Many of the formulae to be set forth are previously known, 

30 wherefore also other equations corresponding to them will be found in the 
literature in the field - for example for calculating the cost function and 
iteration. Yet mathematical consideration is necessary for understanding and 
describing the invention. The vectors are denoted with bold lettering in the 
present study. In this context, the use of two images to determine an optical 

35 flow and a new image are studied, but it is also possible to use a larger 
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number of images, in which case the noise portions appearing in the images 
can be eliminated. 

Determining an optical flow (Figure 1) between two images can be 
represented as an optimization task of two variables, wherein the 
5 interrelation of the variables of the function to be minimized must be 
selected. (There may also be a larger quantity of variables depending on the 
application.) As stated previously, the square sum of the value differences of 
the image points and the smoothness of the flow are normally employed as- 
variables in the gradient-based methods. The aim is thus to minimize the 
10 difference between the values (for example luminosity values) of the image 
points, i.e. pixels, connected by the flow on the one hand, and the 
smoothness of the flow, i.e. the requirement that points adjacent in the first 
image are mapped adjacent to one another in the second image as well, on 
the other hand. 

15 Let a vector field composed of individual motion vectors L (Figure 

3) be formed between the images. The vector field is fixed between the 
images in such a way that the fulcrum k of each vector is midway between 
images a and b. The motion vector fixed at point k = (x, y) intersects the 
image surfaces at points k + t = (x+u, y+v)] and k-t = (x-u, y-v), where t is 

20 the shift of the motion vector from the position orthogonal to the surface of 
the images. 

Let us consider the vector field as continuous, in which case the 
equation to be minimized will be 

25 J = \\f(u(x, y), v(x, y), u x , u y , v x , v y ) dx dy, (1-1) 

where the functional within the integral equation will be 



f= a ( g 1 (x+u(x, y), y+v{x, y)) - g 2 (x-u(x, y), y-v{x, y))f 
+ ( du(x,y) )2 + ( du(x,y) 2 + dv(x,y) 2 + dv(x,y) 2 
dx dy dx dy 

= a(g, (x+u, y+v) - g 2 (x-u, y-v)) 2 + u x 2 + u y 2 + v 2 + v 2 , 

(1-2) 
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where is the value of the pixel of the first image, g 2 is the value of the pixel 
of the second image and u x , u y , v x , v y ave partial derivatives in relation to the 
lower index. 

The functional to be minimized is composed of the smoothness 
5 criterion, i.e. the sum of the partial derivatives of the flow, u x + u y + v x + v y 2 , 
on the one hand, and of the cross-correlation criterion taking account of 
motion, ( g, (x+u, y+v) - g 2 (x-u, y-v)) 2 . on the other hand. The interrelation 
of the criteria is determined by the parameter a. 

The minimum of the functional J can be sought by using the 
10 calculus of variations, by setting the variations of J relative to u and v as 
zero. With this constraint, the following equations are satisfied: 

K-*-JL-±JL = 0 (1-3a) 
du dx du x dy du y 

15 5/_A_^__A^L = 0 d-3b) 

dv dx dv x dy dv y 

The first terms in equations (1-3a) and (1-3b) can be written into 
the following form: 

20 — =2a(g, (x+u, y+v) - g 2 (x-u, y-v)) 

du 

dg^x + u.y + v) | dg 2 (x-u,y-v) ) (14a) 

ax dx 

— =2a(g, (x+u, y+v) - g 2 (x-u, y-v)) 
dv 

dg^x+u.y + v) | dg 2 (x-u,y-v) (14b) 
dy dy 



25 



The latter terms in equations (1-3) can be expressed as: 
" * = 2U „; 1* = 2 U „; (1-5a) 



dx du x dy du 
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d df d df , 

— —-=2v xx - =2v yy . 1-5b 

dx dv x dy dv yy 



By substituting (1-4) and (1-5) in equations (1-3), we obtain: 



a ( g, (x+u, y+v) - g 2 (x-u, y-v)) (^l ( x+u> y+v ) + Ob. ( x -u,y-v)) 

dx dx 

-V 2 u = 0, (1-6a) 



a ( g, (x+u, y+v) - g 2 (x-u, y-v)) (^L ( X+Ui y+v ) + f^i. ( x _ u y _ v ^ 

dy dy 

-V 2 v=0, (1-6b) 

where the Laplace function of the flow V 2 u = u xx + Uyy and V 2 v = v xx + Vyy. 

Depending on what mathematical method of calculation is used, 
the Laplace function can be approximated for example with terms V 2 u = 
u - u, and V 2 v »v - v , wherein 

u (x, y) = 1/4 (u(x,y- 1 ) + w(x-1 ,y) + u(x+1 ,y) + u(x, y+1 ) (1-7) 



where u(x+i, y+i) are the calculation points surrounding point (x, y). The 
same also holds true for v . The number of calculation points in this study is 

20 4, but it may also be e.g. 8. In Figure 7, the four adjacent points surrounding 
point k being considered, said adjacent points being used for the calculation, 
are connected by a broken line. If eight surrounding calculation points were 
used in the calculation, each point surrounding point k which is being 
considered would influence the value of u and v . 

25 The flow can thus be determined by minimizing equations (1-6) by 

means of the following iterative process: 

= u n + V [( u - u„) - a ( flf, (x+u n , y+v n ) - g 2 {x-u n , y-v n )) 
{ lt {X+Un ' Y+Vn) + it {X ~ Un ' V ~ Vn))] (1 " 8a) 

30 
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Vn+i = v„ + y [( v -v n ) -a(g, {x+u n , y+v n ) - g 2 (x-u m y-v n )) 

(x+iv„, y+v n ) + ^ (x-u m y-v„))], (1-8b) 
dy dy 

where y is the relaxation parameter used in the iteration, determining the 
5 proportion of the upgrading term in the new motion vector value. 

Once the flow has been determined in accordance with equations 
1-8, the values, i.e. colours (or the gray scale, for example) of the image 
points in the images between the known images can be determined. The 
values of the image points in the image halfway between the original images 

10 are obtained on the basis of the motion vectors and the intersections of the 
images. When it is desired to define the value of a point between the original 
images that is not situated midway between the images, the motion vector 
passing through this image point must first be determined, whereafter the 
value of the desired point is interpolated from the points of the original 

15 images. Said vector can be sought for instance by means of the following 
iterative algorithm (Figures 4 and 5): selecting the motion vector L0 whose 
fixing point (y,0) is on the same line orthogonal to the images as the point to 
be determined (x, z) (step 51); determining the distance vector eO between 
this motion vector and the point to be determined on the plane having the 

20 direction of the images (step 52); selecting as the new motion vector L1 the 
vector whose fixing point (y1,0) is at the distance indicated by the distance 
vector determined in the preceding step from the fixing point of the previous 
motion vector (y,0) in the reverse direction (-eO) (step 54); repeating these 
steps until the value of the distance vector is considered to be small enough 

25 (step 53). Finally, the value of the new point on the image plane is sought on 
the basis of the motion vector selected (step 55). The motion vector field can 
be made by interpolation to be continuous, that is, the motion vectors 
between the fixing points are formed by interpolating from the values of the 
motion vectors passing through the surrounding fixing points. Hence, motion 

30 vectors passing through the original fixing points or motion vectors 
interpolated between them can be used to form the new image. 

By denoting the co-ordinates of the point to be determined as (x. 
z) and the co-ordinates of the fixing point of the motion vector passing 
through this point as (y, 0), the former can be expressed as: 

35 
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y„ = x-c[t(y^) -z], (2-1) 

where y„ is the new and the previous candidate for the fixing point of the 
motion vector, and t = [u, v]. The coefficient c is a changeable upgrading 
5 coefficient dependent on the earlier upgradings. Let us examine the con- 
vergence of the algorithm (2-1) (presuming that c=1): let us study on what 
conditions to be imposed on the vector field shift the algorithm will yield the 
situation: 

10 y n + t(y n )z = x (2-2) 

that is, when the following holds true: 

lim (y„ + t(y„)-z-x) = 0, (2-3) 

15 

where -1 < z < 1 . The initial error is 

e 0 = x - (y 0 + t(y 0 ) - z ) (2-4) 
20 After the first iteration 

Yi = x - t(y 0 ) - z, (2-5) 
thus giving an error of 



25 



30 



e 1 = x - ( Yl + t(y,) • z) = x - ((x - t(y 0 ) • z) + t(x - t(y 0 ) • z ) • z ) 
= t(y 0 ) • z - t(x - t(y 0 ) • z) • z = z • ( t(y 0 ) - t(x - t(y 0 ) • z )) (2-6) 

Likewise, the errors at rounds n and n+1 will be obtained as 

e„ = x - (y„ + t(y„) • z ) = x - t(y„) z - y n 

e n+1 = x - (y n+J + t(y„ + ,) • z ) = - z • ( t(x - t(y„) • z) - t(y n ) ) (2-7) 
The ratio of the length of the error vectors e„ and e n+7 obtained is 



35 
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|e„J _ \t(x-t(yj-z)-t(y n ) 



5 



25 



z — — - — (2-8) 

x-t(y n ) z-y n I 



By substituting a = x - t(y„) ■ z and b = y n , equation (2-8) can be written into 
the form 

\e^ = 2 \Mltl<M. (2 . 9) 
eJ a-b\ 



Equation (2-9) reaches its minimum and maximum on the surface 
of the images (z = ±1). In order for the algorithm to be convergent with all 
10 values -1 < z < 1 , 

&tm«,. (2 . 10) 

\a-bi 



must hold true. 

15 The lengths of the vectors can be expressed in a variety of ways, 

depending on what mathematical method of calculation is used. Preferred 
methods include the use of the 1-norm or infinite- norm, for example. In the 1- 
norm, the length of the vector is the sum of the lengths of the vector 
components, |t| = |u| + |v|, and in the infinite-norm the length of the vector 

20 is the length of the greatest vector component, 1 1 1 =max{ | u | , I v | }. By using 
the 1-norm, where a=(a.,, a 2 ) and b=(b.,, b 2 ), the left-hand side of equation 
(2-10) can be written into the form 



|t(a)-t(b)| = 
|a-b| 

\u(a„a 2 )-u(b„a,) +|r(a 1 ,a 2 )-y(b 1 , a 2 )| \a,-b,\ 

l a i -b il ! a_b l 



+ \u(a„a 2 )- i#(a v b 2 j + \v(a„a 2 ) - v(a v b 2 \ |a 2 -b 2 | 
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The right-hand side of equation (2-11) corresponds to the 
weighted sum of the difference quotients of vector field t: thus it still holds 
true that 



l*(a)-t(bj f 
la-bl 



du 



dx 



cv 



cx 



\ f 
+ (1-r) 



du 



dy 



dv 



dy 



< 1, 



(2-12) 

where 0 < r< 1. In order for equation (2-12) to be satisfied with all values of r, 



10 



du 



dx 
du 



dy 



+ 



dv 

dx 
dv 



dy 



< 1 



< 1. 



(2-1 3a) 
(2-13b) 



must hold true, where t(x, y) = [u(x, y), v(x, y)]. 

Hence, the algorithm described in equation (2-1) places additional 
constraints on the flow to be sought (equation 2-13), according to which the 

15 absolute value of the difference between the shifts of two neighbouring 
motion vectors must be less than the distance between the fixing points of 
the same neighbouring motion vectors. Since the additional constraints 
pertain to the partial derivatives of the flow, which the smoothness criterion 
seeks to minimize, satisfying the constraints can be guaranteed by reducing 

20 the proportion of the term representing cross-correlation in the upgrading. 
The constraints of equation (2-13) can be realized by replacing parameter a 
in equation (1-8) for example with a parameter obtained from equation 3-1 : 



a(u x , u y , v x , v y ) = cc- max{0, 1 - max { \u x \ + \v x \, \u y \ + \v y \}l6), 
25 (3-1) 



where d is the distance between the fixing points. 

The absolute values of the partial derivatives |u x |, \u y \, \v x \ and \v y \ 
of equation (3-1) can be approximated for instance by means of equations: 

30 

|u x (x, y)\ = max {|u(x+1 , y) - w(x, y)|, \u(x, y) - t/(x-1 , y)|} 



K(*. y)\ = max {|v(x+1, y) - v(x, y)|, \v(x, y) - v(x-1, y)|} 
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\u y (x, y)\ = max {\u(x, y+1) - u(x, y)|, \u(x, y) - u{x. y-1)|} 

I v y (x, y)\ = max {| v(x, y+1)- v(x, y)\ , \ v(x, y) - v(x, y -1 )|} (3-2) 

5 

wherein the differences between the shift of the motion vector and the shifts 
of its closest neighbours are calculated. The greatest difference of shifts 
indicates the smallest distance of the motion vector from the boundaries of- 
the area of movement. Hence, the maximum possible difference of the shifts 

10 equals the distance between the fixing points. There are also other ways of 
approximating the derivative, such as calculating the difference of the shifts 
of the neighbouring motion vectors to the motion vector being studied: |v(x, 
y+1) - v(x, y — 1)|, where the shift of the actual motion vector being studied is 
not used to determine the derivative. 

15 What is essential in equation (3-1) is that the values of a diminish 

when the partial derivatives increase, and that the proportion of the cross- 
correlation, taking account of motion, in the upgrading term vanishes when 
constraints (2-13) are broken (note Figure 2). In other words, when the flow 
is being iterated, for each motion vector its shortest distance to the limit at 

20 which the additional constraints are broken is calculated, and in accordance 
therewith the weighting of the term taking account of motion is upgraded. 
Equations (1-8) can now be written into form 

= u n + rjo (u- u n ) - rjxa ("*, " y . v„ v y ) ( g, {x+u n , y+v n ) - g 2 

25 (x-u n , y-v n )){ (^i (x+u n , y+v n ) + ^ (x-u n , y-v n )) (3-3a) 

dx dx 

Vn+i = v n + Id ( v -v„) - 7x a "y. ^ v y ) ( 9, (x + u n , y+v n ) - g 2 

(x-u n , y-v n )){{^ (x+u n , y+v n ) + {x _ Un> y _ Vn)l (3 . 3b) 
dy dy 

30 where rj D is the unit parameter of the smoothness criterion and rj x is the unit 
parameter of the cross-correlation taking account of motion. 

Figure 6 illustrates the change of parameter a within the permitted 
area of movement MA of the motion vector. Let the value of parameter a be 
c when the motion vector meets the image plane orthogonally, in which 
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situation the motion vector is located in the middle of the area of movement. 
When the shift of the motion vector increases and it approaches the 
boundaries of the area of movement, the value of parameter a decreases, as 
will be seen from the figure. The decrease can be linear, as is shown in the 
5 figure, or non-linear, depending on the application. 

Let us study the additional constraints by means of Figures 7-9. 
Figure 7 illustrates the permitted intersecting area of the motion vector fixed 
to point k with the image plane when the constraining factor is the motion 
vector fixed to a neighbouring fixing point A. Let us denote also this motion 

10 vector A, according to the fixing point. Let us assume A to be known, and 
hence by shifting the motion vector fixed to point A to point k (let us call the 
shifted motion vector A", denoted with a broken line in the figure), the 
constraining influence of A on the permitted shifting area of the motion vector 
fixed to point k is found. The permitted shifting area S (denoted with a 

15 hatched line in the figure) is determined in such a way that the distance of 
the intersection of the image and the motion vector fixed to point k from the 
intersection of the image and motion vector A shall be less than the distance 
between points A and k. When studying the figure, it is to be borne in mind 
that the image plane is different from the plane of the fixing point. In other 

20 words, if the plane of the fixing point is shifted along the motion vectors onto 
the image plane and if all motion vectors have the same direction (in this 
case, the direction of A), a permitted shifting area S is obtained for the 
motion vector being studied that passes through point k when the 
constraining factor is one known motion vector A. Since four surrounding 

25 fixing points have been used in the calculation, the permitted area has the 
shape of a square, at the angles of which the motion vectors passing through 
the neighbouring fixing points are located. The square denoted with a broken 
line illustrates a situation where motion vector A has no shift but the motion 
vector meets the image plane orthogonally. The shape of the area of 

30 movement need not necessarily be a square, but the edges of the square 
may be curved, or the area may even possibly be a circle. At their maximum, 
however, the boundaries of the area of movement must pass through the 
neighbouring points that are used in the calculation. 

Figure 8 illustrates the influence of the motion vectors of the four 

35 closest neighbouring fixing points (A, B, C and D) on the motion vector 
passing through point k. The permitted area S on the image plane is denoted 
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with a hatched line. Figure 9 illustrates the permitted areas of intersection of 
the motion vector on the image planes of the original images. 

First Embodiment of the Invention 

5 The following presents, by way of example, the implementation of 

the invention to a situation where the optical flow between two known images 
is sought (Figure 10) and new images between said images are formed by 
means of the flow and the original images (Figures 4 and 11). 

First, the images to be processed are introduced into the flow 
10 determining system. At this stage, the size of the sampling matrix, i.e. the 
pixel number into which the images are digitalized, is selected. As a next 
step, the value of each pixel defined by the sampling matrix is measured 
from the images (Figure 10, step 91). The value can be for example a colour 
or a grayish hue. What variable is used can be selected in accordance with 
15 the situation. 

Since it is preferred to select the number of points on the fixing 
plane (i.e. the number of motion vectors) to be smaller than the pixel number 
(step 92), the digitalized images are low-pass filtered for example by 
convolution with the Gaussian function (the filtering can be implemented in 
20 some other manner as well, or omitted completely, even though it is 
preferable). The Gaussian function is separable, wherefore the convolution 
can first be calculated in direction x and thereafter in direction y, which is 
more rapid than calculating a bidimensional convolution. Hence, the size of 
the filtering function employed is dependent on the number of motion vectors 
25 (i.e. on the distance between the motion vectors). In connection with the 
filtering, the derivatives and the desired pixel values are stored in memory 
from the surface of both images, for instance the greatest derivatives in the 
direction of each surface coordinate (g' max = max{g' 1i1 , g' 21 , g' 12 , g' 2-2 » and the 
greatest and smallest pixel values between the images (g min = min{g 1> g 2 ,}, 
30 9max = max{g 1 , g 2 ,},). Using these values, the greatest permitted value for unit 
parameter rj x - by means of which the speed of convergence is influenced 
when the desired optical flow is sought - is determined (step 93). 

After the image processing described above, the fixing plane and 
the fixing points therein up to the midpoint of the images are determined 
35 (step 94). The fixing plane may also be at another point than midway, but is 
most advantageous to use a plane formed in the middle. It is to be noted that 
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if the first original image serves as the fixing plane, it suffices to calculate the 
derivatives from the second image only. The number of fixing points is the 
same as the number of motion vectors determined in connection with the 
filtering. The fixing of the motion vectors can also be performed at an earlier 
5 step. 

Thereafter, for each motion vector located at each fixing point (k= 

(x, y)): 

a) The pixel values, the derivatives and the motion vector- 
shift for the images at the intersections of the motion vector and the images 

10 are sought, k+t(x) for the first image and k-t(x) for the second image (step 
95). The derivatives are obtained by calculating the difference between the 
values of two neighbouring image points. The initial direction of the motion 
vector may be directly against the image planes, but if earlier information on 
the direction of the motion vectors is available (for example a motion vector 

15 field calculated chronologically by means of the previous image pair), this 
can be made use of. 

b) The Laplace function of the representation is 
calculated (step 96) by calculating the difference between the motion vector 
shift t(x) and the mean of the shifts of the surrounding motion vectors. The 

20 value of the Laplace function indicates the non-smoothness of the motion 
vector field, i.e., inversely it indicates the smoothness. 

c) The additional constraint (equation 3-1) - i.e. the 
differences between the motion vector shift t(x) and the shifts of the 
surrounding motion vectors - is calculated, and the greatest of these is 

25 selected (step 97). 

d) Thereafter, a new shift t(x) is upgraded for the motion 
vector located at point k = (x, y) (step 98) by substituting the quantities 
calculated above in equations (3-3), in accordance with which the direction of 
an individual motion vector is upgraded as the additional constraint 

30 calculated at step c) changes parameter a. The result is stored in memory if 
the positions of all lines are upgraded simultaneously (Jakobian iteration), or 
alternatively the position of the motion vectors is immediately changed 
(Gauss-Seidel iteration). 

Thereafter, steps a)-d) are repeated 10-15 times or until the 

35 change in the representation is considered to be sufficiently small (step 99). 
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Once the optical flow containing the desired number of motion 
vectors has thus be en achieved, the number of motion vectors can be 
increased, i.e., the resolution of the optical flow can be enhanced (steps 910 
and 911). The initial shifts of the new motion vectors to be added are 
5 obtained by interpolating between the shifts of the previous motion vectors. 
The process is restarted by selecting the number of motion vectors (distance 
from one another) and low-pass filtering. Such processing is continued until 
the desired vector density has been achieved, that is, an optical flow has 
been formed between the images. 

10 A new image (or new images) are calculated by means of the flow 

and the original images using the method represented by equation (2-1) 
(Figures 4 and 5). Another alternative method for forming a new image/new 
images is to interpolate the value of an image point for each discrete motion 
vector of the optical flow to a desired plane z: in other words, the value of a 

15 motion vector point on plane z is obtained by interpolating from the values of 
the intersections of the first image and motion vector and the second image 
and motion vector. Thus, in this method a motion vector point value is 
calculated with the desired value of z (Figure 11). 

Even though the determining of the flow and the new image has 

20 been studied in this example utilizing two original images, it is possible to 
utilize more than two images for determining the optical flow and the new 
image. In such a situation, the calculation and the method must naturally be 
modified from those presented above. 

25 Second Embodiment of the Invention 

The invention can also be made use of in enlarging aliased 
images. Aliased images in this context refer to images in which the sampling 
frequency is small compared to the frequencies contained in the images. 
When aliased images are enlarged, the advantage in accordance with Figure 

30 12 is achieved compared to conventional interpolation, in other words, less 
shadow images will be formed. 

In Figure 12, object A has been enlarged in the orthogonal direction 
by factor two. Object B represents the result obtained by conventional 
interpolating methods, and object C represents the interpolation result 

35 obtained by means of the optical flow. 
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When aliased images are enlarged, the new image is sought 
similarly as in the interpolation between two images: herein, however, the 
new image is sought between two adjacent lines/columns of image points - 
not between two images, but within the image, so to speak. The equations 
5 set forth previously will herein be rendered the following form: 

" n+1 = u n + y((u-u n )-a(u n )(gAx+u„)-g 2 (x-u n )) 

(4k {X+Un ) + *jb- (x-u„)) (4-1) 
dx dx 

and 

10 a{u x ) = - max{0,1 - \u x \}} , (4-2) 

wherein 

|i#J = |u x (x)| = max {|u(x+1) - u(x)|, |u(x-1) - u(x)\}/d, 

(4-3) 

where d is the distance between the fixing points of the motion vectors on 
15 the fixing plane, x is the fixing point of the motion vector, and u is the motion 
vector shift from the orthogonal position. 

Also some other equation than 4-3, approximating the derivative 
at point x being examined, can be used, an example being 

20 |u x | = |u x (x)| = {|u(x+1 ) - |u(x-1 )\ld, (4-4) 

where, therefore, the approximation of the derivative is sought from the 
difference between the shifts of the neighbouring motion vectors and not 
from the difference between the shift of the motion vector of the point being 
25 examined and the shifts of the neighbouring motion vector. Figure 13 shows 
an example of motion vectors formed between two image point lines/columns 
Qi and g 2 . 

Furthermore, the following algorithm can now be used for 
calculating new pixels: 



30 



y n = x - u(y^) ■ z, (4-5) 

where z is the distance of the image plane to be calculated from the 
fixing point plane of the motion vectors. 
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The following is an example, with reference to Figure 14a, of 
implementing the method to a situation where a single image is known, the 
optical flow of each adjacent pixel line and column in the image is sought 
and new pixel lines and columns are calculated by means of the flow and the 
5 original pixel lines and columns. 

First, the images to be processed are introduced into the flow 
determining system. At this stage, the size of the sampling matrix, i.e. the 
pixel number into which the images are digitalized, is selected, unless the 
images are already rendered in digital form. As a next step, the value of each 

10 pixel is measured from the images (Figure 131). The value can be for 
example a colour or a grayish hue. What value is used can be selected in 
accordance with the situation. 

The number of motion vectors is selected, and the two first pixel 
lines are low-pass filtered for example by convolution with the Gaussian 

15 function (step 132). In connection with the filtering, the greatest derivatives in 
both directions of the image plane (g' max = max{g'., i> S , 2.t 9\.i' 9 2.2 }) ar) d tne 
greatest and smallest value differences between the images (for example 
gray scale values g min = minfer,. gj, g max = max^, g 2 }) are stored in memory. 
The derivatives are obtained by calculating the difference between the 

20 grayish hue of two adjacent image points. Parameter Oq is set in such a way 
that the greatest possible upgrading shift of the lines will be as great as 
possible but will not violate the constraint, e.g. = d I (g' max • (g max - 
g min ))(step 133). 

After the image processing described above, the fixing plane and 
25 the fixing points therein are determined up to the midway of the pixel lines 
(step 134). The number of fixing points is the same as the number of motion 
vectors determined in connection with the filtering. The fixing of the motion 
vectors can also be performed in an earlier step. 

Each motion vector (fixed to each fixing point) is submitted to the 
30 following procedure: 

a) The values of the image points, the motion vector shift and the derivatives 
at the intersections of the motion vector and the pixel line are sought, x+u(x) 
in the first line and x-u(x) in the second line (step 135). 

b) The Laplace function, i.e. the smoothness of the representation is 
35 calculated for example by calculating the difference between the shift of the 
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motion vector u(x) and the mean of the shifts of the motion vectors flanking it 
(step 136). 

c) The additional constraint is calculated (step 137), that is, e.g. the 
differences between the shift of the motion vector u(x) and the shifts of the 

5 adjacent motion vectors are determined and the greatest of these is 
selected. 

d) The quantities are substituted in equation (4-1), and thus the additional 
constraint changes parameter a and the motion vector shift is upgraded (step 
138). 

10 Steps a)-d) are performed for each motion vector, and they are 

repeated 10-15 times, or until the change in the representation is considered 
to be sufficiently small (step 139). 

Thereafter, the number of motion vectors is increased if desired 
(steps 1310 and 1311) and the process is restarted with a new low-pass 
15 filtering. The initial shifts of the new lines to be added are obtained by 
interpolating between the shifts of the previous lines. The process described 
above is repeated until the desired line density has been achieved. 

Once the flow between the two first pixel lines has been formed 
and stored, the next pixel line pair is proceeded to (Figure 14b, steps 1312 
20 and 1313). The same operations are performed on the new pixel line pair as 
on the first line pair, except that the previous second line now becomes the 
first line, which is why the filtering must be repeated for the latter line only. 

Each pixel line pair in the image is processed in the manner 
described above, whereafter the method proceeds to processing pixel 
25 columns (steps 1314 and 1315). Also the pixel column pairs in the figure are 
similarly processed. 

The desired new pixel lines and columns are calculated by means 
of the flow and the original pixels by either of the methods described above 
(equation 4-5, Figure 4 or Figure 11). 
30 To save memory space, the new pixel lines and columns can also 

be calculated immediately subsequent to the determination of the flow, and 
thus the storing of the flow is replaced by forming new pixel lines/columns. 

The flow required to enhance the resolution can also be formed by 
calculating the pixel lines and columns simultaneously: A bidimensional flow 
35 matrix is formed from the motion vectors to be calculated e.g. in accordance 
with the attached Figure 15. The fixing points of the vectors corresponding to 
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the lines are denoted with R and the fixing points of the vectors calculated on 
the basis of the columns are denoted with S. The fixing points are 
interconnected with elucidatory lines in the figure. The empty points are 
points in the original image. 
5 In this connection, a rectangular set of pixel co-ordinates has 

been examined, but the pixel rows could just as well be at an angle of 45°, 
between which rows the new pixels are formed. In this context, it is also 
possible to use more than two pixel rows/columns for determining the flow 
and the new image. 

10 In comparison with the prior art, the invention affords a fairly short 

calculation time on account of the fact that there is no need to calculate each 
intermediate image separately. The invention also enables considerable 
changes between images on account of the suppression of the motion- 
minimizing term as the constraints are broken at the occlusion points. 

15 Moreover, the method is very robust on account of automatic adjustment 
(change of weighting factor). 

The invention has very wide application. The motion information of 
the optical flow can be used for example for interpolating or extrapolating 
new images, improving the quality of an image sequence, and speed 

20 measurement by means of a camera. The determination of corresponding 
image points also has bearing upon forming three-dimensional images on 
the basis of stereo image pairs. Motion recognition based on images is used 
in several applications of computer vision. Motion information can also be 
used for altering images in an image set and for reducing, i.e. compressing, 

25 the information required for transmitting an image set. Since by means of this 
method an unlimited number of new images can be formed using one flow, 
image compression is more effective than in the prior art methods. 

The fact that realization of the constraints is attended to enables 
the use of a rapidly convergent and computationally light algorithm for 

30 determining the flow, which makes it advantageous to implement the 
invention also in applications to which the production of new images does 
not directly relate. 

The method can also be used for enhancing image resolution and 
producing high-quality still frames from video images. 

35 Even though the invention has been described in the above 

mainly by means of examples of an image to be formed between two original 
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images and enlargement of an aliased image, the invention can be 
implemented in other embodiments within the scope of the inventive idea; for 
example more than two images can be used for determining the flow and 
forming a new image. 

5 
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Claims 

1 . A method for forming a motion vector field composed of motion 
vectors between at least two images in connection with image processing, in 
which method the motion vector field is formed by minimizing it in relation to 

5 at least two criteria, the first criterion representing the motion information 
between the images and the second criterion the smoothness of the motion 
vector field, 

characterized in that additional constraints, determining motion 
vector-specifically the limit values of the directional change permitted to the 

10 motion vector, are used in forming the motion vector field, and that in forming 
the motion vector field the weighting of the first criterion is adjusted motion 
vector-specifically to be smaller when the limit values are approached, and 
thus on account of the adjustment the proportion of the first criterion is 
suppressed and the proportion of the second criterion increases when the 

15 motion vector field is formed by minimization. 

2. A method as claimed in claim 1, characterized in that 
when the motion vector reaches the limit values of the permitted directional 
change, the weighting of the first criterion is zeroed. 

3. A method as claimed in claim 1, characterized in that 
20 the weighting of the first criterion is proportional to the smallest distance of 

the motion vector from the limit values, according to which the absolute value 
of the difference between the directional change of two adjacent motion 
vectors, wherein the directional change corresponds to a deviation from the 
direction orthogonal to the image plane, said directional change being 
25 expressed as a shift vector having the direction of the image plane, shall be 
smaller than the distance between the fixing points of the same motion 
vectors, wherein the fixing points define the location points through which the 
individual motion vectors shall pass. 

4. A method as claimed in claim 1, in which method: 

30 - the number of motion vectors and their distance from one 

another are selected, 

- the value and derivatives of each pixel are determined, 

- a unit parameter is set for iterating the motion vector field, 

- for each motion vector, the values and derivatives of the 
35 intersections thereof and the image planes are sought, 

- the smoothness of the motion vector field is calculated, 
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- for each motion vector, the magnitude of the change of its shift 
vector having the direction of the image plane, the shift vector 
corresponding to the deviation of the motion vector from the 
direction orthogonal to the image plane, is determined, 

5 - a motion vector field is iterated, so that the change of the shift 

vectors of the motion vectors is sufficiently small, 

- the number of motion vectors is increased, 

- the above steps are repeated until the number of motion vectors 
is sufficient, 

10 characterized in that the motion vectors of the motion 

vector field are fixed to a plane in the middle of the images, said fixing 
defining for each motion vector a point through which that motion vector 
must pass, and limit values are sought for the shift vector of each motion 
vector, and said limit values are used for iterating the motion vector field. 

15 5. A method as claimed in any one of claims 1 to 3, 

characterized in that the motion vectors of the motion vector field are 
fixed to fixing points that are located on a plane in the middle of the images. 

6. A method as claimed in any one of claims 1 to 4, 
characterized in that the motion vectors of the motion vector field are 

20 fixed to fixing points that are located on the plane of one of the images. 

7. A method as claimed in any one of claims 1 to 4, 
characterized in that the motion vectors of the motion vector field are 
fixed to fixing points that are located on a plane between the images. 

8. A method as claimed in claim 3 or claim 4, 
25 characterized in that the limit value of the directional change of each 

motion vector, wherein the directional change is expressed as a shift vector 
having the direction of the image plane, said shift vector simultaneously 
defining thn motion vector point on the image plane, is obtained by 
determining the differences between the shift vector of each motion vector 
30 and the shift vectors of the surrounding motion vectors, and selecting of 
these the one having the greatest absolute value. 

9. A method as claimed in claim 3 or claim 4, charac- 
terized in that the limit value of the directional change of each motion 
vector, wherein the directional change is expressed as a shift vector having 

35 the direction of the image plane, said shift vector simultaneously defining the 
motion vector point on the image plane, is obtained by determining the 
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differences between the shift vectors of the motion vectors surrounding each 
motion vector, and selecting of these the one having the greatest absolute 
value. 

10. A method as claimed in any one of the preceding claims, 
5 wherein a new image is formed from original images by means of a motion 

vector field, characterized in that for each predetermined pixel of the 
new image a motion vector passing through said pixel is first sought, and a 
value is interpolated for each pixel of the new image from the values of the 
pixels of the original images, said values being obtained from the 
10 intersections of the selected motion vector and the original images. 

11. A method as claimed in any one of claims 1 to 9, wherein a 
new image is formed from original images by means of a motion vector field, 
characterized in that a value is interpolated for each intersection of 
the motion vector and the desired new image plane, said intersection 

15 defining the location of the pixel in the new image, from the values of those 
pixels of the original images which are obtained from the intersections of 
each motion vector and the original images. 

12. A method as claimed in claim 10, characterized in 
that the motion vector passing through a pixel of the new image to be 

20 determined is sought by initially selecting a motion vector, defining the 
distance vector between the motion vector being sought and the pixel to be 
determined on a plane having the direction of the images, selecting as the 
new motion vector a vector whose fixing point is at the distance determined 
by the distance vector defined above but in the reverse direction from the 

25 fixing point of the motion vector previously selected, and repeating the 
selection of a new motion vector until the value of the distance vector is 
considered to be sufficiently small, the motion vector last selected being the 
motion vector sought. 

13. A method as claimed in claim 1 or claim 4, charac- 
30 terized in that when an odd number of original images is used, the 

motion vectors are fixed to the plane of the original image in the middle. 

14. A method for forming a motion vector field between pixel 
lines/columns of an image in connection with image processing, 
characterized in that in the method, the motion vector field is solved 

35 by minimizing it in relation to at least two criteria, the first criterion 
representing the motion information between the pixel lines/columns and the 
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second criterion the smoothness of the motion vector field, additional 
constraints determining motion vector-specifically the limit values of the 
directional change permitted to the motion vector being used in forming the 
motion vector field, and that when a solution to the motion vector field is 
5 sought the weighting of the first criterion is adjusted motion vector- 
specifically to be smaller when the limit values are approached, wherein on 
account of the adjustment the proportion of the first criterion is suppressed 
and the proportion of the second criterion increases when the motion vector 
field is formed by minimization. 
10 15. A method as claimed in claim 14, characterized in 

that when the motion vector reaches the limit values of the permitted 
directional change, the weighting of the first criterion is zeroed. 

16. A method as claimed in claim 14, characterized in 
that the weighting of the first criterion is proportional to the smallest distance 

15 of the motion vector from the limit values, according to which the absolute 
value of the difference between the directional change of two adjacent 
motion vectors, wherein the directional change corresponds to the deviation 
from the direction orthogonal to a pixel line/column, said directional change 
being expressed as a shift vector having the direction of a pixel line/column, 

20 shall be smaller than the distance between the fixing points of the same 
motion vectors, wherein the fixing points define the location points through 
which the individual motion vectors must pass. 

17. A method as claimed in any one of claims 14 to16, 
characterized in that in the method: 

25 -the number of motion vectors and their distance from one another 

is selected, 

- the value and derivatives of each pixel are determined, 

- a unit parameter is set for iterating the motion vector field, 

- for each motion vector, the values and derivatives of the 
30 intersections thereof and the pixel lines/columns are sought, 

- the smoothness of the motion vector field is calculated, 

- for each motion vector, the magnitude of the change of its shift 
vector having the direction of its pixel line/column, the shift vector 
corresponding to the deviation of the motion vector from the 

35 direction orthogonal to the pixel line/column, is determined, 
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- a motion vector field is iterated using the limit values of the shift 
vectors, so that the change of the shift vectors of the motion 
vectors is sufficiently small, 

- the number of motion vectors is increased, 

5 - the above steps are repeated until the number of motion 

vectors between pixel lines/columns is sufficient, 

- the above steps are repeated for each pixel line/column pair. 

18. A method as claimed in any one of claims 14 to 17, 
characterized in that the motion vectors of the motion vector field are 

10 fixed to a straight line in the middle of the pixel lines/columns. 

19. A method as claimed in any one of claims 14 to 17, 
characterized in that the motion vectors of the motion vector field are 
fixed to a straight line between the pixel lines/columns. 

20. A method as claimed in any one of claims 14 to 17, 
15 characterized in that the motion vectors of the motion vector field are 

fixed to a pixel line/column. 

21. A method as claimed in claim 16 or claim 17, 
characterized in that the limit value of the directional change of each 
motion vector, wherein the directional change is expressed as a shift vector 

20 having the direction of the pixel line/column, said shift vector simultaneously 
defining the motion vector point in the new pixel line/column, is obtained by 
determining the differences between the shift vector of each motion vector 
and the shift vectors of the adjacent motion vectors, and by selecting from 
these the one having the greatest absolute value. 

25 22. A method as claimed in claim 16 or claim 17, 

characterized in that the limit value of the directional change of each 
motion vector, wherein the directional change is expressed as a shift vector 
having the direction of the pixel line/column, said shift vector simultaneously 
defining the motion vector point in the new pixel line/column, is obtained by 

30 determining the difference between the shift vectors of the motion vectors 
flanking each motion vector. 

23. A method as claimed in any one of claims 14 to 22, wherein 
the pixel number in an image is changed by means of a motion vector field, 
characterized in that for each predetermined pixel of a new pixel 

35 line/column in the image to be changed, a motion vector passing through 
said pixel is selected, and for each pixel of the image to be changed, a value 
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is interpolated from the values of the pixels of the original images, said 
values being obtained from the intersection of the selected motion vector and 
the pixel lines/columns of the original image. 

24. A method as claimed in any one of claims 14 to 22, wherein 
5 the pixel number in an image is changed by means of a motion vector field, 

characterized in that for each intersection of a motion vector and a 
straight line of the desired new pixel line/column, said intersection defining 
the location of a new pixel, a value is interpolated from the values of the 
pixels of the original images, said values being obtained from the intersection 
10 of each motion vector and the pixel lines/columns of the original image. 

25. A method as claimed in claim 23, characterized in 
that the motion vector passing through a pixel being determined in the new 
pixel line/column in the changed image is sought by first selecting a motion 
vector, determining the distance vector between the motion vector being 

15 sought and the pixel to be determined on a straight line having the direction 
of the pixel line/column, selecting as the new motion vector a vector whose 
fixing point is at the distance determined by the distance vector defined 
above but having a reverse direction from the fixing point of the motion 
vector previously selected, and repeating the selection of a new motion 

20 vector until the value of the distance vector is considered to be sufficiently 
small, the motion vector selected last being the motion vector sought. 

26. A method as claimed in any one of claims 14 to 25, 
characterized in that more than two original pixel lines/columns are 
used. 

25 27. A method as claimed in claim 26, characterized in 

that when an odd number of original pixel lines/columns is used, the motion 
vectors are fixed to the straight line of the middlemost of the original pixel 
lines/columns. 
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